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ABSTRACT 


Two training squadrons of student naval aviators flying the T-2C Buckeye aircraft 
during their Intermediate Stage of Strike (Jet) Training were compared to identify any 
effects attributed to exposure to the T-45 Operational Flight Trainer, a state-of-the-art, 
visual, dome simulator. Students in VT-23 at Naval Air Station (NAS) Kingsville, Texas 
were exposed to demonstrations of carrier landings in the T-45 simulator prior to their 
carrier qualifications. A sister squadron at NAS Meridian, Mississippi (VT-19) did not 
receive exposure to the simulator. Both squadrons conducted carrier qualifications on the 
same dates and same aircraft carrier. A comparison of the squadrons considered the four 
measures of effectiveness (MOE): 1) squadron disqualification rates, 2) LSO trend 
analysis scores, 3) Aviation Training Form (ATF) CQ-11x performance evaluations, and 
4) average grades for individual performance areas listed on ATF CQ-11x. Students in | 
VT-23 had higher CQ-11x performance evaluations. When the performance evaluations 
are broken down to individual performance areas, VT-23 also has substantially higher 
grades in Pattern. Allowing for the importance that a proper flight pattern has on the 
approach and landing on the aircraft carrier, it is argued that exposure to the T-45 visual 


simulator had a positive effect on the students in VT-23. 
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EXECUTIVE SUMMARY 


With the introduction of a new training jet, the T-45A Goshawk, the Navy is 
introducing an entirely new training program. The T-45 training program is heavily 
computerized, emphasizing computer based training and a state-of-the-art, visual, dome 
simulator. When Hughes designed the new training jet, they designed more than just an 
aircraft, they designed an entire training system, including classroom lectures, computer 
based training, and two highly advanced, state-of-the-art simulators. The most advanced 
of these flight simulators introduces visual simulation into the training pipeline. 

The T-45 Operational Flight Training (OFT) simulator is built around a replica of 
a Goshawk cockpit. The visual simulation is projected onto a partial dome screen, 
allowing the student a 180 degree horizontal and 60 degree vertical field of view. The 
three areas in the training curriculum that require the greatest use of the Student Naval 
Aviator's (SNA) visual skills are landing skills, weapons training and carrier 
qualifications. The visual system allows extensive training in all of these areas. For 
weapons training, the simulator includes air/ground bomb, racket and strafing models, 
rocket models, and the dust puffs of ordnance exploding on the ground at the impact 
points. For landing skills, several highly detailed models of Naval Air Stations, outlying 
fields and Air Force Bases are also modeled. Finally, for carrier operations, a dynamic 
model of the aircraft carrier USS Nimitz is provided. In the simulator, the SNA can 
experience carrier deck arrested landings, on deck taxiing, and catapult takeoffs in sea 
states from zero to sea state five. Rain, fog, cloud cover, sea state, and turbulence can 
be introduced, modified, or dismissed by simple instructor-computer interface controls in 
order to provide a more realistic aircraft flight deck environment. 

Such innovation in training, not to mention the cost of such innovation, begs the 
question of effectiveness. How effective is the training program? How effective is the 
visual simulator over the standard instrument simulators? The U.S. Navy Jet 
Undergraduate Training Program (JUTP) is conducted in two locations, Meridian, 
Mississippi (VT-19) and Kingsville, Texas (VT-23). The T-45 visual-dome simulator was 
installed in Kingsville in 1990. From 1990 to 1993 the T-45 Goshawk was not ready for 


X11 





students. Instructor training for the T-45 began in the summer of 1993 and students 
started actual T-45 flights nearly a year later. However, while the simulator waited in 
Kingsville, T-2C Buckeye students were exposed to demonstrations in the flight simulator 
prior to their carrier qualifications and weapons training. The T-2C students in Meridian 
did not receive visual simulator exposure. By comparing these two groups in Meridian 
and Kingsville it may be possible to quantify and hopefully analyze the effectiveness of 
the new visual simulator. 

Sixty-nine students from VT-19 and 67 students from VT-23 were reviewed for 
this study. Both squadrons conducted carrier qualifications on the same dates and same 
aircraft carrier during May and July of 1993. VT-19 used five different Landing Signal 
Officers (LSO) over this period. VT-23 used seven different LSO's. Measures of 
effectiveness (MOE) for the comparison were 1) squadron disqualification rates, 2) LSO 
trend analysis scores, 3) Aviation Training Form (ATF) CQ-11x average performance 
evaluations, and 4) average grades for individual performance areas listed on ATF CQ- 
llx. The MOE's were evaluated using a regression model that tested for squadron, 
month, LSO, and interaction effects. 

The resulting analyses indicate that there was no difference between the squadrons 
— in disqualification rates and LSO trend analysis scores. A significant difference exists in 
the CQ-11x average performance evaluations. VT-23 appears to have a higher average, 
albeit over a very small range. When the average performance evaluations are broken 
down to individual performance areas, the difference between the squadrons is more 
acute. VT-23 has substantially higher grades in Pattern, and moderately higher grades in 
Start Position, Power Control, and Glideslope Control. Ruling out possible effects due 
to differences in the LSO's, the Pattern grade remains as the most significant difference 
between the squadrons. 

While it must be argued that the measures of effectiveness considered are 
subjective measures of human evaluation, the importance that a proper pattern has on the 
entire carrier landing process cannot be argued. Although students are taught to make 
most of their turns during the landing pattern using the cockpit instruments, they are also 


encouraged to use visual landmarks on the ground while learning the approach. These 
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visual aids are not available to the student during a carrier approach, and while the 
student 1s constantly reminded of this fact, it must certainly be disorientating during the 
first few approaches to the ship. One advantage of the T-45 visual simulator, may be an 
awareness of this spatial disorientation. In the simulator, the student is given an 
opportunity to observe the relatively small deck of a carrier against a massive, featureless 
ocean, and has the opportunity to witness a simulated instrument approach to the carrier. 
_ Such awareness may encourage students to place more emphasis on their pattern, thus 
resulting in higher grades. | 

Due to the limited nature of this study, it is imprudent to make a blanket statement 
about the effectiveness of the T-45 visual simulator. However, although further study is 
warranted, the simulator appears to have a positive effect on student's ability to recognize 


and fly the proper pattern as they approach the aircraft carrier. 
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I. INTRODUCTION 


A. A BRIEF HISTORY OF THE USE OF SIMULATORS IN AVIATION 


Since the inception of flight, the aviation community has looked for better, safer 


methods to train new pilots with minimal loss of students and aircraft. Flight simulators 


allow an instructor to present many of the skills required to control an aircraft while the 
student is still on the ground. Simulator use is recorded as early as 1910 in the United 
States, Britain, and Italy, an indication of the importance of this new form of training. 
Early simulators were actual modified aircraft mounted on universal joints that, under 
proper wind conditions, allowed the student to control for air turbulence. Other inventors 
engineered manual levers for the instructor to manipulate the simulator's movements. The 
common technique stressed by these early simulators was movement--the students ability 
to manually control the aircraft. 

In 1929, Edwin Link designed a pneumatically controlled simulator. Link's 
simulator still required pitch, roll, and yaw movements to be initiated manually, but then 
used a pneumatic bellows to fine tune the movement. The Link simulator was announced 
to have simulated the true feel of flying without ever leaving the ground. (Rolfe and 
Staples 1986, 19-20). But just as Link's patent cleared, flight stmulation began to take 
another direction. Instructors began to realize that simulator motion was not an effective 
training aid. As aircraft became more involved, instruments were installed on the 
simulators to mimic the flight controls on the more modern aircraft. Simulators proved 
to be excellent instrument trainers, an important task that they still perform today. 

By the beginning of World War II, military air corps in Europe, the United States, 
and Japan were conducting basic instrument training in Link simulators. During World 
War II, simulators were used for much more than training new pilots. Simulators were 


modified to enhance celestial navigation and bombing skills. World War II also saw 








simulators being adapted to train for specific types of aircraft. By the end of World War 


If, the aviation world realized that they depended more on the simulator's instrument — 


capabilities and less on the movement of the simulator. Post World War II development 
stressed electronic and analog instrument trainers as the flight simulator's primary mission. 

Today, flight simulators come in many forms, from simple static cockpits that allow 
pilots to practice ejection techniques to the most advanced F/A-18 trainer that incorporates 
motion, pressure suits and 360-degree visual, dome simulation. With the advances in 


computer generated graphics, visual flight simulators provide a degree of reality never 


before experienced in flight simulation. A pilot can now experience the feeling of three- 


dimensional flight without ever leaving the ground. 


B. THE SIMULATOR AND U.S. NAVAL AVIATION TRAINING 

Throughout the development of flight simulation, the military has been at the 
forefront in its support and use of the technology. The military realized the potential of 
the aircraft's fighting capability during World War I. At this time, the military also 
recognized the need for rapidly teaching flying skills to large numbers of men (Rolfe and 
Staples 1986, 16). While simulators were not used extensively during World War I, the 
need to optimize training resources and student qualification inspired the development of 
the discipline of Aviation Psychology. Aviation Psychology would not only develop 
guidelines for the military's selection of recruits for flight training, but advise the 
engineers who built the simulators used in their training. 

During World War II, the U.S. Navy trained with a simulator built by Bell 
Laboratories (Rolfe and Staples 1986, 28). The Bell Laboratories simulator consisted of 
a PBM-3 front fuselage and cockpit, with complete controls, instrumentation and auxiliary 
equipment, and even included a computing device to solve flight equations. The PBM-3 
simulator is credited as the first operational flight trainer that attempted to simulate the 
characteristics of a particular aircraft. 

Today the United States Navy has a flight simulator for every aviation platform it 


flies. In addition to real flight hours, pilots are required to fly simulated flights on a 














regular basis. Student pilots begin using flight simulators in basic flight school and 
continue into their specialized training in either the Strike (Jet), Fixed Wing, or Rotor 
communities. In the flight training pipeline, instrument training is stressed. Most pilots 
do not fly a visual simulator until they complete their training program and report to their 


first squadron. 
1. U.S. Navy Jet Undergraduate Training Program (JUTP) 


In the past, the training pipeline for jet pilots consisted of three distinct stages using 
three different aircraft. The Primary stage was common to naval aviators, basic flight 
training in the twin turbo-prop T-34. Once basic flight training was completed, SNA's 
assigned to the jet training pipeline began the Intermediate stage of their instruction. The 
Intermediate stage used the T-2C Buckeye aircraft. This stage lasted for approximately 
8 months and concluded with initial carrier qualification. From the Intermediate stage the 
Student Naval Aviator (SNA) progressed to the Advanced stage and yet another aircraft, 
this time the A-4 Skyhawk. In the Skyhawk, the SNA practiced many of the skills 
learned previously in the T-2C. Upon completion of the Advanced stage, usually about 
10 months, the student qualified for carrier landings for the second time. Successful 
completion of all three stages certified the SNA as fully qualified, "winged" Navy jet 
pilot. The SNA was assigned to a specific Navy platform (F-16, A-6, F/A-18) and 
transferred to that aircraft's Fleet Readiness Squadron (FRS). While training simulators 
of all types were used in the Jet pipeline, the simulators stressed instrument work and 


provided no visual simulation. 


2.  T-45 Goshawk Traming Program 

With the introduction of a new training jet, the T-45A Goshawk, the Navy is 
introducing an entirely new training program. The T-45 training program is heavily 
computerized, emphasizing computer based training and a state-of-the-art, visual, dome 
simulator. The T-45A Goshawk aircraft was designed to eliminate the Intermediate stage 
of the jet training pipeline. New SNA's will now fly only one platform after their 
Primary Stage. The T-45 curriculum is designed to last approximately 1 year and covers 


all of the skills taught in the original training pipeline, with one important difference, only 





one carrier qualification 1s now required. When Hughes designed the new training jet, 
they designed more than just an aircraft, they designed an entire training system, 
including classroom lecture, computer based training, and two highly advanced, state-of- 
the-art simulators. The most advanced of these flight simulators introduces visual 
simulation into the training pipeline. 

The T-45 Operational Flight Training (OFT) simulator is built around a replica of 
a Goshawk cockpit. A Gould SEL 32/8780 computer with 2M bytes of memory controls 
the simulation scenario. While anchored to a fixed platform, the simulator uses G-seat/G- 
motion cueing systems and an aural system to provide realistic aircraft motion, audio and 
aerodynamic cues. The visual simulation is projected onto a partial dome screen, 
allowing the student a 180 degree horizontal and 60 degree vertical field of view. The 
three areas in the training curriculum that require the greatest use of the SNA’s visual 
skills are landing skills, weapons training and carrier qualifications. The visual system 
allows extensive training in all of these areas. For weapons training, the simulator 
includes air/ground bomb, racket and strafing models, rocket models, and the dust puffs 
of ordnance exploding on the ground at the impact points. For landing skills, several 
highly detailed models of Naval Air Stations, outlying fields and Air Force Bases are also 
modeled. Finally, for carrier operations, a dynamic model of the aircraft carrier USS 
NIMITZ is provided. In the simulator, the SNA can experience carrier deck arrested 
landings, on deck taxiing, and catapult takeoffs in sea states from zero to sea state five. 
Rain, fog, cloud cover, sea state, and turbulence can be introduced, modified, or dismissed 
by simple instructor-computer interface controls in order to provide a more realistic 


aircraft flight deck environment (T-45A Simulator Training System Overview, 1-3 - 1-4). 


C. VISUAL SIMULATOR EFFECTIVENESS 

| The training effectiveness of visual dome simulators is hard to measure. One means 
of measuring simulator effectiveness is through transfer of training: the ability of the 
student to transfer what he has practiced in the simulator to the actual aircraft (Lintern 


et al. 1990, 320). But the skills trained in a simulator are controlled and constrained by 











the simulator's characteristics. These characteristics include realism of the cockpit and 
instrument panel, realism of motion effects, and in the case of visual simulators, realism 
of the visual system. | | 

Even if transfer of training 1s verified, singling out the individual role played by a 
simulator characteristic can be difficult. At the University of Illinois, researchers found 
that beginning flight students were positively affected by visual simulator landing practice 
prior to solo landings in actual aircraft (Lintern et al. 1990, 324). While this study 
indicates that visual simulators have a positive training effect, how much of the training 
transfer can be attributed to the simulator's visual system is hard to measure. Student 
motivation and the additional training with similar controls may also contribute to the 
improved flight performance (Gopher et al. 1994, 401). In an attempt to isolate the effect 
of a visual system, various studies have examined how much visual acuity is required 
to achieve a transfer of training. Detail and field of view may also play an important 
part in the transfer of training. Density, or the visual definition of a simulated object, 
appears to be more important than the visual detail of the object (Kleiss and Hubbard 
(1993, 653). Concurrent with the development of the T-45 simulator and training 
program, D. P. Westra and colleagues conducted a study for the Naval Training Systems 
Center in Orlando, Florida (Westra et al. 1985). The Westra study provided key 
recommendations for the design of carrier landing tasks for the T-45 visual simulator. 
In the study one group of SNA's received training in an experimental visual simulator and 
were compared against a control group of SNA's who did not receive training in the 
simulator. Special equipment was developed to measure aircraft deviations as the student 
approached the landing field. Westra's study is important because it identified a 
measurable improvement in Flight Carrier Landing Practice (FCLP) performance when 
student pilots practiced with the simulator. However, Westra did not extend his research 
to the deck of an actual carrier because of difficulties with stabilizing his equipment on 


a rolling deck. 








D. PROBLEM STATEMENT 

The U.S. Navy JUTP is conducted in two locations, Meridian, Mississippi and 
Kingsville, Texas. The T-45 visual-dome simulator was installed in Kingsville in 1990. 
From 1990 to 1993 the T-45A Goshawk was not ready for students. Instructor training 
for the T-45A began in the summer of 1993 and students started actual T-45A flights 
nearly a year later. However, while the simulator waited in Kingsville, T-2C Buckeye 
students were exposed to demonstrations of the flight simulator prior to their carrier 
qualifications and weapons training. The T-2C students in Meridian did not receive visual 
simulator exposure. By comparing these two groups in Meridian and Kingsville, this 
study attempts to quantify and analyze the effectiveness of the new visual simulator. 
Chapter II describes in detail carrier qualification requirements, the data, and the measures 
of effectiveness used for the study. The results of the data analysis are presented in 
Chapter III. Discussion of the results and conclusions of the study are addressed in 


Chapters IV and V, respectively. 





It. METHODOLOGY 


A. CARRIER QUALIFICATION REQUIREMENTS 

Student Naval Aviators reviewed for this thesis were in the intermediate phase of 
their strike flight training. Previous training included basic ground school and basic flight 
skills in the T-34 turbo prop training aircraft. Students receive carrier qualification 
instruction as the last section of their intermediate syllabus. Previous sections in the 
syllabus include Familiarization, Basic Instruments, Radio Instruments, Airways 


Navigation, Basic Formation, Night Fly, Gunnery/Weapons, and Out of Control Flight. 
1. Field Carrier Landing Practice (FCLP) 


During carrier qualification training, the student receives 11 periods of instruction 
which are coded CQ-1 through CQ-11x. CQ-1 begins with a basic review of procedures 
and flight skills to allow both the student and the instructor to adapt to each other. The 
student is assigned to the same instructor throughout the CQ stage. The instructor is a 
qualified Landing Signal Officer (LSO) trained and certified to assist pilots as they 
approach and land on the deck of a carner. During CQ-1 through CQ-10, the student 
flies Field Carrier Landing Practice (FCLP) landings on an air strip, which is painted to 
resemble the deck of an aircraft carrier. A Fresnel Lens Optical Landing System 
(FLOLS), identical to the FLOLS on a carrier, is used to help the student line up 
properly.’ The instructor/LSO guides the student to the airstrip using the same directions 
that will be used on the aircraft carrier. At the end of CQ-1, CQ-2, CQ-9, and CQ-10, 
the student is evaluated by his instructor using an Aviation Training Form (ATF). A grade 
is assigned based, in part, on an evaluation of the student's headwork, airwork, start 
position, line up, and speed control. A full listing of the performance areas considered 
on these ATF's is presented in Appendix A. Grades are given on a four point interval 


scale with 1 = unsatisfactory, 2 = below average, 3 = average, 4 = above average. The 


' The FLOLS is sometimes referred to as the glideslope indicator or in pilot 
vernacular "the ball". 





points are summed and averaged for a final ATF score. At the end of CQ-10, the student 
is evaluated for field qualification. If field qualification is granted, the student advances 


to CQ-11, which includes the actual carrier landing. 


2. Carrier Qualification | 

For his carrier qualification, the Student Naval Aviator receives two numeric grades, 
an overall flight performance grade. and a LSO trend analysis grade of his actual 
approaches and arrests on the carrier deck. The LSO/instructor evaluates the Student 
Naval Aviator's performance in nineteen areas listed on the Aviation Training Form 
(ATF) for CQ-11 as either Unsatisfactory, Below Average, Average, and Above Average. 
While the LSO's evaluation of these performance areas is subjective, his own flight 
experience as well as standardized instructor training, lend validity to the assigned grade. 

In addition to the specific items listed on the CQ ATF, the LSO must also consider 
the following criteria for qualification: 

a Student displays no dangerous tendencies. 

a | Student demonstrates steady or improved performance during FCLP/ship 
qualification period. 

s Student requires minimum LSO assistance during the final two 
approaches/landings. 

a Student is predictable-ready for Advanced Strike. 

# Student has a 50 percent or better boarding rate (Chief of Naval Air Training 
(CNATRA) CQ instruction 1994, III-3). | 

The second numerical grade is reported on the LSO Trend Analysis Sheet. Every 
pass and approach that a Student Naval Aviator makes on the carrier is graded by his 
LSO. This practice is continued even after the SNA completes his training and enters a 
Fleet Squadron. The student is required to make at least six approaches to the aircraft 
carrier; four of the approaches must be complete, arrested landings and two of the 
approaches must be touch and go landings. Each approach to the carrier is assigned a 
grade on a five point scale. Once again the grades are subjective but are based on the 


LSO's experience observing and waving other pilot approaches. Westra writes that 











previous research has identified four factors that the LSO considers when assigning the 
trend analysis scores. These four factors, in order of importance are: 1) touchdown 
accuracy, 2) approach glideslope control, 3) approach lineup control, and 4) approach 
angle of attack control (Westra et al. 1985, 41). The LSO marks and grades are listed in 
Appendix B. The minimum grade acceptable for the LSO Trend Analysis is 2.4. 
Qualification of students with less than 2.4 may be given in special cases, but only if the 
student demonstrates an overall improving trend. Additionally, a student with 2.4 or 


better may be disqualified for not meeting any of the criteria scored on the ATF. 


B. DATA DESCRIPTION 
- Data were collected for SNA's at the CNATRA archives at Naval Air Station (NAS) 
Corpus Christi, Texas for the carrier qualification periods in May and July of 1993. 
1. Boat Detachment Dates 
Carrier qualification flights were conducted between May 14-19 and July 23-27, 
1993. USS AMERICA (CV 66) and USS JOHN F. KENNEDY (CV 67) were the 
designated carriers for May and July, respectively. Both carriers were located off the 
eastern coast of Florida. Students and their instructors flew their aircraft from Key West, 
Florida and met the carriers at sea. The weather during these qualification periods 
appeared mostly mild, with sea state ranging from 1/2 to 2 meters, winds 5-22 knots, and 
a cloud ceiling averaging around 3500 feet (Fleet Numeric Meteorological and 
Oceanographic Center, 1993). A summary listing of the weather conditions is provided 
in Appendix C. 
2. Training Squadrons 
a. VT-19, NAS Meridian, Mississippi 
- The students at VT-19 did not receive exposure to the T-45 simulator. 
Student scores from the May and July carrier qualification periods of 1993 were analyzed. 
Of the 79 students who participated in the two Carrier Qualification periods, 69 student 


records were available for examination. 





b. VT-23, NAS Kingsville, Texas 

Documentation indicates that 75 T-2 students in Kingsville received exposure 
to the T-45 simulator prior to their carrier qualifications during two separate carrier 
qualification periods in May and July of 1993. Exposure to the simulator consisted of a 
90 minute demonstration of carrier approaches, arrests, and touch and goes. Each student 
sat in the cockpit of the simulator for approximately 5-10 minutes and observed the 
aircraft fly from approximately three miles aft of the carrier, through the approach pattern, 
to the deck for a touch and go, and back around the pattern again for an actual arrested 
landing. Students did not actually fly the simulator because the controls did not match 
those of the A-4, but the students could hold the stick and feel the aircraft movements and 
power adjustments. The instructor monitoring the demonstration could stop the simulation 
at any time for questions or observations. Sixty-seven student records were available. 
A breakdown of the number of students from each carrier qualification period and 


squadron 1s listed in Table 1. 


Table 1: Number of students participating during each carrier qualification versus 
number of students reviewed. 


Reviewed Reviewed 


3. The Landing Signal Officer (LSO) 











For their carrier qualification instruction, students are assigned to a Landing Signal 
Officer (LSO), a pilot certified to provide directions to other pilots during their carrier 
approach and landing. LSO's are trained in accordance with the LSO NATOPS. While 


assigned to his/her first squadron, a pilot may begin the LSO qualification process. First 


10 











the pilot attends a three week LSO school in Oceana, Virginia. Upon completion of the 
school, the pilot returns to his squadron and begins an on-the-job-training qualification 
process, beginning with the Field Qualification. The Field Qualification allows the pilot 
to direct other pilots flying his own type of aircraft to land on an airfield when a fully 
qualified LSO is present. The second qualification, the Squadron Qualification, allows 
the pilot to land any type of aircraft on the deck of an aircraft carrier during periods of 
good weather. The final qualification, the Wing Qualification, permits the pilot to land 
any aircraft on the deck of an aircraft carrier, during day or night. When the pilot 
transfers from the squadron to a training squadron, he/she must be either Squadron or 
Wing Qualified as an LSO. The pilot attends a three day refresher LSO school and must 
attend at least three carrier qualification periods before being designated Training LSO. 

In the training squadron, the LSO instructs, evaluates, and grades the student 
through out the entire carrier qualification period. Should a student disqualify during the 
carrier qualification period, he is reassigned to a new LSO and must repeat all previous 
carrier qualification lessons. The students examined in this study were praded by 12 
different LSO's, 5 LSO's from VT-19 and 7 LSO's oni VT-23. A breakdown of the 
LSO's and the number of their students reviewed are listed in Table 2. To protect 


_ confidentiality, LSO's are referred to alphabetically. 


C. MEASURES OF EFFECTIVENESS 

In comparing the two squadrons, VT-19 and VT-23, several measures of 
effectiveness (MOE) were used to determine if the T-45 visual simulator had an effect 
on SNA carrier qualification. The first MOE looked at the percentage of disqualifications 
within each squadron. The second MOE examined the final LSO Trend Analysis grade 
for each SNA. The third MOE examined the final ATF performance evaluation for each 
SNA. Finally, the CQ-Ilx ATF was broken down into its separate 19 performance areas 


to determine if there was a difference in certain areas of flight performance. 








Table 2: LSO breakdown by squadron and carrier qualification period. 


Squadron Month Number of Students 
VT-19 VT-23 | May July May July 
xX XX 


x 9 


7 
8 
7 
7 
8 
9 
6 
9 
9 
8 
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Ii. RESULTS 


A. COMPARISON OF DISQUALIFICATION RATES 
Table 3 presents the number of disqualifications (DQ) per squadron during each 


carrier qualification period. 


Table 3: Disqualification rates by squadron and carrier qualification period. 


May July 
Squadron | Number Number Percent Number Number Percent — 
Qualified Disqualified Disqualified | Qualified Disqualified Disqualified 


VT-19 36 2 5.2 


10.0 
VT-23 49 | 3 7.0 29 3 9,3 





To compare the two squadrons, a two by two contingency table was developed by 
counting the number of disqualifications and qualifications within each squadron during 
a particular qualification period. A contingency table is used to count the number of 
occurrences and compare that count to the expected number of occurrences based on the 
assumption that the DQ rate is the same for both groups. The difference between the 
actual and expected occurrences is used to generate a chi-square number. Large chi- 
square values indicate large differences between the actual and expected occurrences. For 
this comparison, four two by two contingency tables were used. Two tables were used 
to test the hypothesis that the number of disqualifications was independent of the 
squadron during the qualifications periods of May (Table I) and July (Table II). The two 
remaining tables were used to test the hypothesis that the number of disqualifications was 
independent of the qualification period for VT-19 (Table III) and VT-23 (Table IV). The 
final contingency table (Table V) pools squadrons and qualification period together into 
a two by four table, again testing for independence in the number of disqualifications. 
Table 4 lists the results of the five contingency tables. Judging from the very low chi- 


square values and corresponding high p-values, there is no evidence to say that the 
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number of disqualifications is independent of squadron and carrier qualification month. 
The squadrons appear to disqualify equal number of students. Complete contingency 


tables are presented in Appendix D. 


Table 4: Contingency Table results for comparison of disqualification (DQ) 
counts. 











Contingency Table Tested for Independence DF — Chi-Square p< 
I May DQ's l 0.131 0.7174 

II July DQ's l 0.443 0.5057 

II] VT-19 DQ's l 0.441 0.5066 

IV VT-23 DQ's ] 0.143 0.7053 

All DQ's 3 0.786 0.8528 





B. COMPARISON OF LSO TREND ANALYSIS SCORES 

The LSO evaluates every approach the student flies on the carrier using a seven 
point scale labelled from zero to five, where zero indicates a cut or an unsafe pass and 
five indicates a perfect pass. Table 5 lists the grading scale used by the LSO for the 
carrier approach evaluation. Students are required to make at least six passes (four arrests 
and two touch and gos). At the end of the qualification period the student's approach 
grades are averaged to two decimal places to form the student's LSO Trend Analysis 
Score. Four of the students reviewed for this study did not have a complete LSO Trend 
Analysis form in their training record. Their LSO Trend Analysis records are not 


reviewed in the following comparison. 


Table 5: LSO Trend Analysis grading scale | 
Definition Grade 













OK Perfect pass. 5 
OK _ Reasonable devations with good corrections. 4 
(OK) Fair, reasonable deviations. 3 3 
B Bolter. Zio 
- Below average but safe pass. 2 
PWO _ Power waveoff 2 
OWO Own waveoff. Z 
WO Waveoff | 1 
Cut. Unsafe, Gross deviations made inside waveoff window. 0 
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The LSO's from a given squadron graded only students from that squadron. 
Additionally, each student was graded by the same LSO during the entire carrier 
qualification process. Because 12 different LSO's graded students during this time period, 
consideration must be given to individual grading differences between the LSO's. Table 
6 presents the breakdown of the LSO's used during a qualification period by squadron. 
Note that not all the LSO's grading in May, graded in July. Likewise additional LSO's 
were present in July who did not grade in May. In constructing a model for comparison 
of the two squadrons, it is necessary to make assumptions about the LSO's employed in 
grading SNA's. The analyses presented are essentially based on the concept of a pool of 
trained LSO's, any of whom might be assigned to a particular squadron (and set of 


students). 


Table 6: Breakdown of LSO's used by each squadron during a particular 


qualification period. 
Squadron VT-19 VT-23 Month Total 


May LSO's A,B,C,D, EJF,G,H,I,J 10 
July LS O's A,B,C,E F.G,H,I,K,L 10 
5 7 


Squadron Total 















Figure 1 presents box plots for the LSO trend analysis scores given by individual 
LSO's during May and July qualification periods. The box plots in this graph present 
important information on the distribution of the LSO's trend analysis scores. The top and 
bottom line of the box represent the first and third quartiles of the range for the LSO's 
trend analysis grades. Fifty percent of the data is located within the box. The line within 
the box is the median trend analysis score given by that LSO. In some cases, the LSO 
gave the same grade to several students and the median trend analysis score may be 
located on the top or bottom of the box. Median values are labeled on the boxes to aid 
in their recognition. Lines extending from either end of the box indicate the shape of the 


_ distribution's tail. Long lines indicate a stretched distribution, short lines indicate 
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compactness and bunched data. Asterisks represent outliers, data points that differ 
significantly from the rest of the data (Chambers et al. 1983, 22). The width of each box 
plot is proportional to the square root of the sample size of the data used to plot the box. 
To aid in the interpretation of sample size, the actual sample size appears as a number 
beneath each box plot. 

As can be seen from Figure 1, there appears to be very little difference in the 
median trend analysis score given by the LSO's during the May qualification period. 
Medians range from approximately 2.43 for LSO C to 2.81 for LSO J. The relatively 
short median range of 0.38 indicates little variance between the LSO's. However, VT-19 
has three low outliers and VT-23 has a one low and one high outlier. In all cases, a low 
outlier corresponds to a student who ultimately was disqualified during that qualification 
period. The minimum and maximum median for July are 2.5 for LSO's E, F, and I and 
2.65 for LSO L. Both bounds occur for LSO's in VT-23. However, the range of medians 
in July is considerably smaller than the range of medians in May, 0.15 vice 0.31. The 
lack of outliers in July is probably due to the data reviewed rather than an actual grading 
difference. Students who disqualified in July were given a second opportunity to qualify 
in September. Records were not reviewed for September, hence the July LSO Trend 
Analysis Sheet recording the disqualification were not reviewed as well. 

When the trend analysis scores are combined across both qualification periods, 
Figure 2 results. Figure 2 illustrates the trend analysis scores for the LSO's. The median 
range extends from 2.44 to 2.81. VT-23 has two additional outliers, one high and one 
low, for a total of four outliers. VT-19 still has the three low outliers. For the most part 
however, there appears to be little variance in the LSO's grading procedures. This 
observation can be seen more clearly in Figure 3, which presents box plots of the 
squadron's trend analysis scores over both qualification periods. Even with the presence 
of the low outliers in VT-19, the box plots are remarkably similar. The overall median 
trend analysis score for the two squadrons is only .01. As indicated by the above figures, 


there appears to be little difference in the LSO trend analysis scores in VT-19 and V-23. 
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LSO Trend Analysis, Both Qualfication Periods 









Trend Analysis Score 





Figure 2: Box Plots of LSO Trend Anlaysis Scores, Both Qualification 
Periods. 
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Figure 3: Box Plots of Squadron LSO Trend Analysis Scores, Both 
qualification periods. 
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To quantify the box plot comparisons, an Analysis of Variance (ANOVA) was_ 
conducted to test for significant differences between the squadrons. An ANOVA can be 
used to test for the equality of two or more means that come from independent, normal 
populations with equal variances. It should be noted however, that the primary use of an 
ANOVA is to model data from a designed experiment in which certain factors are 
controlled while others vary. As the data for this study is historical, the ANOVA model 
may oversimplify any relationships in the data. 

As mentioned earlier, the particular LSO's assigned to a given squadron at a given 
time have received the same training and are selected haphazardly, with no special 
purpose. (The same is true of the student aviators.) The statistical model employed for 
the following ANOVA discussions is aimed at identifying possible differences between 
the two squadrons undergoing carrier qualification; any identified difference may then be 
attributed to the use of the simulator. The individual trend analysis score received by any 
student is assumed to be an accumulation of effects of (a) the squadron trained in 
(simulator or no simulator), (b) the month of qualification (May or July), (c) possible 
interaction between squadron and month, and (d) the particular LSO grading the given 
student. In statistical jargon, the squadron and month contributions are fixed effects, 
while the LSO's are random effects and nested within squadrons. An implication of these 
assumptions then is that (possible) differences in the fixed effects are judged relative to 
the variability between the LSO's within the squadron (as opposed to the variability within 
scores issued by the same LSQ). 

The currently available statistical packages do not allow for nested factors with 
varying jumbers of levels (in May 5 LSO's were used in each squadron, while in July 
there were 4 LSO's used by VT-19 and 6 LSO's used by VT-23). The ANOVA results 
in Table 7 were computed in APL and (partially) verified by the MINITAB® program. 


The complete model is presented in Appendix E. 
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The results confirm the prediction of the box plots. Squadron and qualification 
period have no significant effect on the LSO trend analysis scores. The hypothesis that 
the mean trend analysis scores are equal between the squadrons and qualification period 
cannot be rejected. Likewise, there appears to be no significant interaction between the 
squadrons and qualification period. While there does appear to be some variance between 
the LSO's within a squadron, the variance is not statistically significant with a = 0.05. 
If the exposure to the visual simulator had any effect on the students in VT-23, its effects 


are not apparent in the the trend analysis scores. 


Table 7: ANOVA Results for LSO Trend Analysis Scores. 


Effect 7 Sum Squares Mean Squares F-value p-value 


Squadron 0.1943* 0.1943 2.41 0.151 
Qualification Period 0.03 10* 0.0310 0.39 0.549 
Interaction 0.0246* — 0.0246 0.31 0.592 


LSO Variance 0.8049* 0.08049 1.72 0.066 
Error 5.2556 0.0454 
Total 6.4548 

* Adjusted 





C. COMPARISON OF LSO PERFORMANCE EVALUATION 

In addition to the Trend Analysis, the LSO also grades the student on 19 different 
performance areas listed on ATF CQ-1ix. These performance areas consist of headwork, 
procedure, airwork, pattern, start position, speed control, attitude control, power control, 
line up, glideslope control, error detection/correction, waveoff technique, touch and 
sorbolter technique, response to LSO, formation/pattern entry, radio procedures, fuel | 
- management, deck procedures, and catapult procedures. Each of these 19 areas receives 
a mark of unsatisfactory, below average, average, or above average. The marks are then 
converted to an ordinal scale with "1" at unsatisfactory and "4" at above average. The 
LSO performance evaluation is the average of the grades assigned to the 19 performance 
areas. LSO performance evaluations will be compared in a similar manner to the LSO 


trend analysis scores. 
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Figure 4 presents box plots for the LSO performance evaluations given by 
individual LSO's during May and July qualification periods.” During the May 
qualification period, it is readily apparent that the LSO's in VT-19 gave lower 
performance evaluations than the LSO's in VT-23. The range of the median is 2.89 to 
3.05, with the low median in VT-19 and the high median in VT-23. Within the 
squadrons, both VT-19 and VT-23 have a median range of 0.105. Such a small range 
seems to indicate little difference between the individual LSO's. Of the ten LSO's, five 
have a median of 3.0, yet another indication of similarity. However, the distribution of 
the performance evaluations across all LSO's and across the individual squadrons appear 
quite different. In spite of the five 3.0 medians, the majority of performance evaluations 
in VT-19 occur below 3.0 and the majority of performance evaluations in VT-23 occur 
above 3.0. The July qualification presents a similar picture. The range of the median 
extends from 2.95 to 3.05. Within the squadrons, there appears more similarity between 
the LSO's, both in their median performance evaluations and the distribution of the 
grades. Note that of the ten LSO's who graded during this period, six had median 
performance evaluations at 3.0. However, VT-19 still appears to have lower performance 
evaluations than VT-23. The majority of VT-19 performance evaluations occur below 3.0 
and the two lowest medians occur in this squadron. VT-23 performance evaluation 
appear more equally distributed about 3.0. In fact of the six LSO's in VT-23, four have 
medians at 3.0, and the other two LSO's are split about the 3.0 mark. 

The difference between the squadrons is more pronounced when the performance 
evaluations are examined over both qualification periods, as illustrated in Figure 5. The 
range of the median extends from 2.89 to 3.05. While a range of 0.16 seems relatively 
insignificant, the distribution of the grades, belies the small range. Seventy-five percent 
of the grades, for all LSO's in VT-19, are at or below 3.0. In VT-23, the grades are more 


appear more equally distributed about 3.0. Figure 6 presents box plots of the squadron 


*Recall that two additional students were reviewed for this comparison. Hence, the 
sample sizes will not match the sample sizes for the May LSO Trend Analysis 
Comparison. 
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Figure 5: Box Plots of Individual LSO Performance Evaluations, Both 
qualification periods. 


Squadron CQ-11X Performance Evaluation 


© 
sS 
< 
P) 
e 
aed 
oO 
An 





Figure 6: Box Plots of Squadron Performance Evaluations, Both 
qualification periods. 
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performance evaluations. The difference in this graph is even more pronounced. With the 
espeption of the three outliers, all of the grades in VT-23 are at the median of 3.0.> The 
grades in VT-19 have greater variation and are centered on the median of 2.95. Figures 
4-6 seem to indicate that there is difference in the CQ-11x performance evaluations 
between VT-19 and VT-23. Notably, students in VT-23 appear to have higher 
performance evaluations. | | 

An ANOVA similar to the model used for the LSO trend analysis was developed 
to further examine the difference between the squadron performance evaluations. The 
hypothesis to be tested assumes that the mean performance evaluation is equal between 
the squadrons and qualification period and that there is no interaction between squadron 
and qualification period or extreme variance between the LSO's. Results of this analysis 
are presented in Table 8. As predicted by the box plots, there is a significant difference 
between the squadrons. However, there is no significant difference between the 
qualification periods nor is there any interaction between the squadrons qualification 
periods. Between the LSO's there is a significant difference in the variation of their 


grading procedures. Because the squadron's used their own LSO's, it impossible to 


Table 8: ANOVA Results for CQ-11x Average Performance Evaluations. 


Effect Sum Squares Mean Squares’ F-value p-value 
Squadron 1 0.0736* 0.0736 12.23 0.006 
Qualification Period 1 0.0056* 0.0056 0.93 0.358 
Interaction l 0.0016* 0.0016 0.26 0.619 
LSO Variance 10 0.0601* 0.0601 3.12 0.001 
Error 122 0.2353 0.0019 

Total 136 0.3653 

* Adjusted 















‘While it appears that all but three performance evaluations is 3.0, the previous box 
plots have shown that this is not the case. The outliers in this graph correspond to more 
than one, and in most instances, several grades at 2.95, 3.05, and 2.10. However, as the 
vast majority of the grades in VT-23 were at 3.0, the distribution of the grades is 
unimodal to the extreme. 
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implicitly state that the difference in the performance evaluations was caused by the 
visual simulator. The difference may also be attributed to individual squadron differences 


and grading trends. Further examination of the performance evaluations is required. 


D. COMPARISON OF INDIVIDUAL PERFORMANCE AREAS 

Recall that the CQ-11x performance evaluation 1s the average of 19 separate 
performance areas. Figure 7 presents the average performance grade of both squadrons 
for each of the 19 areas evaluated on ATF CQ-11x. From the graph, VT-19 appears to 
have significantly lower performance scores in Pattern, Start Position, Power Control, 
Error Detection/Correction and Glideslope Control, and significantly higher performance 
scores in Waveoff Technique, Touch and Go/Bolter Technique, and LSO Response. With 
some subtle differences, the squadrons appear to grade similarly during both qualification 
periods. (LSO's in VT-19 grade somewhat higher during July in the areas of Waveoff 
Technique, T&G/Bolter Technique, and LSO Response, and some what lower in the areas 


of Power control than they graded in May.) 


LSO Peformance Area Averages 
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Figure 7: Squadron Performance Area Averages, Both qualification periods. 
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The data used to generate Figure 7 come from an ordinal scale of values "1", "2", 
"3" and "4". For all 19 performance areas, in both squadrons, the median value is "3", 
with numerous ties. Quantifying the differences identified in Figure 7 may well be a 
futile exercise, but two nonparametric tests are available for the processing of such data. 
The Median Test tests the hypothesis that all squadrons have the same median against the 
alternate hypothesis that the medians are different. As with the chi-square test, the 
Median Test requires an expected value of one in each cell to insure accuracy (Conover 
1980, 172). Table 9 lists the results of the Median Test for each of the 19 performance 
areas. In most cases, data equal to three are counted as equal to or above the median. 
However in such cases where there are few or no data points below the median, data 
values equal to three are counted as below the median. Grades in Attitude Control, Error 
Detection/Correction, Waveoff Technique, Touch and Go/Bolter Technique, and Response 
to LSO are counted using the latter procedure. 

Results of the Median Tests reject the equal median hypothesis for the performance 
areas of Pattern, Start Portion, Power Control, Glideslope Control, Touch and Go/Bolter 
Technique and Response to LSO. These six performance areas correspond to six of the 
eight areas identified in Figure 7. The two remaining areas, Error Detection/Correction 
and Waveoff Technique, did not fail the Median Test. It is interesting to note, though ‘ot 
surprising by any means, that the number of values above and below the combined 
median correspond to the averages illustrated in Figure 7. Thus, VT-19's Pattern average, 
much lower than VT-23, is reflected in the 25 values "below the median' in Table 9, the 
highest count in this category. Likewise, VT-19 also had the highest average in LSO 
Response, and again, with a count of nine, has the highest count in the ‘above the median’ 


category. (Recall the rules for counting "3" as above or below the median.) 
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The Median Test is a robust test, especially suited for data with expected outliers 
or error. Even though most of the performance grades in the squadrons are "3", whether 
grades of "1", "2", and "4" can be considered outliers is questionable. Consequently, a 
stronger nonparametric test was used to back up the results of the Median Test. The 
Kruskal-Wallis Test offers a nonparametric alternative to a one-way analysis of variance 
and tests the hypothesis that the squadron's grades for a specific performance area are 
identically distributed against the hypothesis that they are not identically distributed and 
their means are different (Conover 1980, 230). For this comparison, the performance area 
grade was used as the response variable and squadron was used as the factor. The 
Kruskal-Wallis Test ranks the data with "1" to the smallest data point, "2" to the next 
smallest, and so on to the largest data point. The ranks are then used to compute the test 
statistic. There is some disagreement surrounding how ties should be handled during the 
ranking process. Conover (1980, 232) advocates averaging the assigned ranks, so if the 
data has four values of "3", and "3" would have been the fourth rank, then the four values 
are initially ranked "4", "5", "6", "7" and then averaged so that all four have the rank a): 
Lehmann (1975, 20) argues that such averaging makes the test invalid in data sets with 
large numbers of ties. He proposes an adjusted test statistic that divides the original test 
statistic by a factor that considers the number of tied data points. As pointed out in 
previous discussion, the performance area grades have numerous ties, most notably about 
the grade of 3. Because of the large number of ties, it is felt that Lehmann’s adjusted test 
statistic 1s more relevant for the Kruskal-Wallis Tests that follow. 

Table 10 presents the results of the Kruskal-Wallis Tests used to examine the 
individual performance areas. While the adjusted test statistic will be used for further 
analysis, p-values computed by both test statistics are listed in the table. The squadrons 
had different performance area means in six performance areas: Pattern, Start Position, 


Power Control, Glideslope Control, Touch and Go/Bolter Technique and Response to 
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LSO. These are the same six areas identified by the Median Test. It is interesting to 
note that without the adjusted test statistic, the only significant performance area is 
Pattern, which is also the performance area showing the largest difference between 
squadrons in Figure te: 

It is clear from the above Kruskal-Wallis tests that students in VT-19 had lower 
grades in the four performance areas of Pattern, Start Position, Power Control, and 
Glideslope Control and higher performance grades in Touch and Go/Bolter Technique and 
Response to LSO. At this point in the analysis, it would seem appropriate to look at 
individual LSO difference, to determine if differences within a squadron contributed to 
the differences between the squadrons. Figures 8 and 9 illustrate the performance area 
averages of individual LSO's within VT-19 during the May and July qualification periods, 
respectively. In each of the graphs, if the all the LSO's gave an average of "3" in a 
performance area, that area has been removed from the graph. | 

In Figure 8, it can be seen that LSO A grades high when the other LSO's grade low 
and has a much lower grade in Glideslope control than any other LSO in the squadron. 
At first, glance VT-19 seems to have less variability in July (Figure 9) than it did during 
May. LSO A grades more similarly to the other LSO's, albeit somewhat higher in 
Attitude Control and Line Up. LSO C has a much higher grade in Response to LSO than 
the other LSO's. To examine the five LSO's in VT-19 more closely, Kruskal Wallis 
Tests, similar to the models used above, examined each performance area pictured in 
Figures 8 and 9. The Kruskal-Wallis Tests were conducted over both qualification 
periods, using the LSO's grades from both periods. Results of these tests are listed in 
Table 11. Italicized rows indicate a performance area that tested positively for significant 
difference between the LSO's, p-value < 0.05. Table 11 illustrates the variability between 
the LSO's in VT-19. Seven of the twelve performance areas examined have significant 
differences between LSO's. Of the six performance areas noted in Table 10, only Pattern 


was graded consistently by the LSO's. 
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VT-19 LSO Peformance Area Averages, May 


4 —o 4 
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VT-19 LSO Performance Area Averages, July 
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Figure 9: VT-19 Performance Area Averages, July. 
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Table 11: Results of Kruskal-Wallis Test on grades in selected performance 
areas by LSO's in VT-19, both qualification periods. 





z 
Rs 


Error DetectiorCorrection 
Waveoff Technique 
T&G Bolter Technique 
Response to LSO 


4 
4 
f 
4 
4 
4 
4 
4 
4 
4 
4 


Figures 10 and 11 illustrate the average performance grades in VT-23 during May 
and July qualification periods, respectively. Compared to VT-19, VT-23 displays much 
less variability. In May (Figure 10), no LSO stands out as an individual, although LSO 
J has higher averages in the areas of speed control and attitude. At first glance, Figure 
11 looks disjointed and appears to suggest great variability between VT-23's LSO's 
during July. However, closer examination reveals that except for Line Up and Glideslope 
Control, only one LSO grades differently from the other five LSO's. 

The lack of variability revealed in Figures 10 and 11 is reflected in the results of 
the Kruskal-Wallis Tests used to examine differences between the LSO's in VT-23 listed 
in Table 12. Only Attitude Control and Response to LSO display significant differences 
between the LSO's. 
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*VT-23 LSO Perfromance Area Averages, May 
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Figure 10: VT-23 Performance Area Averages, May. 


VT-23 LSO Performance Area Averages, July 
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Figure 11; VT-23 Performance Area Averages, July. 
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Table 12: Results of Kruskal-Wallis Test on grades in selected performance 
areas by LSO's in VT-23, both qualification periods. 





6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 


Thus comparison of LSO's within a squadron indicate considerable variability in 
VT-19 and somewhat less difference in VT-23. Of the six areas identified in Figure 7 
and confirmed in Table 9 at the beginning of this chapter, only grades in Pattern can be 
considered unaffected by difference between the LSO's within a squadron. The 
differences in Pattern between the squadrons, appear to be just that, differences between 


the squadrons. 
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IV. DISCUSSION 


Comparison of students in VT-23 to VT-19 revealed no significant differences in 
disqualification rates or LSO trend analysis scores. However, students in VT-23 had 
higher performance evaluations than students in VT-19. Moreover, the differences in 
performance evaluations appear to emphasize differences primarily in the area of Pattern. 
As Figure 7 and subsequent nonparametric tests indicate, VT-23 had higher grades in 
Pattern than students in VT-19. Isolating the cause of VT-19's higher Pattern grades, 
however, is a difficult task. 

As has been discussed in previous chapters, LSO grades are somewhat subjective. 
Westra argues in his study that LSO grades are too subjective to form any concrete 
results. Westra points out that LSO's sometimes use grades as a motivational factor to 
encourage students to perform better in subsequent phases of their training, thus 
decreasing objectivity and validity (Westra et al. 1985, 41). However, Westra's study 
concentrated on the Field Carrier Landing Practice (FCLP) portion of the carrier 
qualification process and he studied only the LSO Trend Analysis Scores, not the CQ 
performance evaluations. (FCLP's are recorded on ATF's CQ-1x through CQ-10x.) When 
he considered the actual carrier qualification, the LSO grades for students who used the 
simulator were somewhat lower than the grades for students who did not use the 
simulator, but the difference was not statistically significant. In this study, students with 
exposure to the simulator had somewhat higher LSO trend analysis scores than students 
who did not have exposure, but the difference was statistically insignificant as well. 

Examination of the CQ-11x Performance Evaluation Averages revealed significant 
differences, albeit across a small range, between students in VT-23 and VT-19. Further 
analysis identified the most significant difference between the squadrons appeared in the 
_ grades for Pattern. Nonparametric tests also identified significant differences in Start 


Position, Power Control, Glideslope Control, Touch and Go/Bolter Technique, and 
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Response to LSO. However, when these five performance areas are examined within 
individual squadrons, significant differences exist between the LSO's. Pattern is the only 
performance area without significant difference between LSO's within the squadrons. 
The question still remains as to the cause of the difference in the Pattern Grades. 
Because each squadron used different LSO's, it is impossible to rule out grading 
differences between the squadrons. However, as the carrier qualification is the last 
requirement for the carrier qualification stage of their training, there 1s no longer the 
‘motivational requirement for grades suggested by Westra, therefore some of the 
subjectivity may be ignored. Indeed, previous research supports the claim that part of the 
difference in Pattern grades may be attributed to exposure to the visual simulator. Both 
Lintern and Westra found that use of a visual simulator improved their subject's spatial 
awareness. Lintern used a visual simulator to train beginning civil flight students before 
the landing phase of their instruction. He suggests that approximately 2 hours of landing 
training in a visual simulator can reduce the number of pre-solo landings in an aircraft 
(Lintern et al. 1990, 324). While Lintern admits to the possibility of instructor bias in 
determining the correct number of pre-solo landings, he attempted to counter such bias 
by not telling the instructors that the number of pre-solo landings would be the measure 
of effectiveness for the study. Westra found a significant advantage in approach lineup 
control for students who had trained on a visual simulator (Westra et al. 1985, 49). These 
findings were supported using equipment observations as opposed to human evaluation. — 
For his study Westra used a special device that recorded the pilot's approach to the air 


field and recorded deviations from the correct line up approach. 


A. THE IMPORTANCE OF PATTERN 

At sea, Pattern is the element that drives a pilot's entire approach to the carrier. If 
the Pattern is correct, then the Line Up, Power Control, and Glideslope Control should 
follow closely behind. In the Flight Training Instruction Manual for the T-2C, it states: 
"The landing pattern is the key to successful ball flying. This cannot be emphasized 


enough. Your primary goal during the early stages of FAM [familiarization stage] is to 
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fly a consistent, precise, pattern resulting in consistent starts. When you are able to do 
so, then you can think about flying the ball (CNATRA P-650, 1993, 3-10 - 3-11)." 
Students begin practicing the proper landing problem in the second stage of their 
instruction. (Recall that the Carrier Qualification Stage occurs during the ninth and final 
stage of Intermediate flight instruction.) Figure 12 illustrates the standard approach 
pattern flown by students throughout their instruction. Although students are taught to 
make most of their turns using the cockpit instruments, they are also encouraged to use 
visual landmarks on the ground while learning the approach. These visual aids are not 
available to the student during a carrier approach, and while the student is constantly 
reminded of this fact, it must certainly be disorientating during the first few approaches 
to the ship. One advantage of the T-45 visual simulator may be an awareness of this 
spatial disorientation. In the simulator, the student is given an opportunity to observe the 
relatively small deck of a carrier against a massive, featureless ocean, and has the 
opportunity to witness a simulated instrument approach to the carrier. If two hours of 
visual simulator practice was enough to reduce pre-solo landing flights, then possibly two 
hours of simulator exposure is enough to enhance a student's awareness of the importance 
of instruments in flying the proper pattern. Such awareness might encourage students to 


place more emphasis on their pattern, thus resulting in better performance. 


B. AREAS FOR FURTHER STUDY 

At the completion of this thesis, approximately 121 students will have completed 
some portion of the T-45 training program, either by flying T-45A aircraft for their entire 
training or by transitioning to the T-45A for Advanced training after flying the T-2C for 
Intermediate training. As more T-45 aircraft are built, the T-2C and A-4 training 
squadrons in Meridian, Mississippi will eventually be phased out and all Naval jet flight 


training will be conducted in the T-45 aircraft. Until this phase out is | 


“With the exception of those pilots designated to fly the E-2/C-2 Hawkeye Electric 
Reconnaissance Aircraft. 
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Instruction: Intermediate Strike and Advanced E-2/C-2, September 1993, 
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complete, however, the opportunity exists to compare the T-45 squadron in Kingsville to 
the A-4 squadron in Meridian. The T-45 training program has the ability to gather and 
record detailed data on students in the program. With prior planning, a similar recording 
system could be established for the A-4 program. Granted, the use of different aircraft 
introduces variance to the comparison, but such variance should not nullify all 
observations. In the long term, T-45 students should be, and undoubtedly are being, 
tracked as they progress beyond the training squadron. The integration of the T-45 
students into the FRS and Fleet Squadrons should be compared to the integration of the 
A-4 students. Does the training program help the new aviators transition to their new 
platforms? 

The emphasis of these comparisons must be on the training program. The T-45 
training program is such a radically different approach to training, incorporating 
traditional lectures with high tech computer based training and visual simulators, it would 
seem impossible to isolate the effect of the visual simulator from the other training aides. 
However, as the simulator plays such an important role in the program, its effects will not 


be ignored. 
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V. CONCLUSION 


This thesis compared two T-2C training squadrons to see if exposure to the T-45 
visual simulator had any effects on student naval aviators during carrier qualifications. 
Student Naval Aviators in VT-23 received exposure to the T-45 simulator. SNA's in 
VT-19 did not train with the simulator. Both squadrons completed their final carrier 
qualifications concurrently, flying on the same aircraft carrier. Four measures of 
effectiveness were examined: Qualification Rates, LSO Trend Analysis Scores, Average 
CQ-11X Performance Evaluations, and grades in the Individual Performance Areas 
considered by CQ-11X. Comparison of the squadrons found no difference in qualification 
rates or LSO Trend Analysis Scores and significant differences in Average Performance 
Evaluations and in some Individual Performance Areas, most notably Pattern, Start 
Position, Power Control, Glideslope Control, Touch and Go/Bolter Technique, and 
Response to LSO. As this study compared grades based on human evaluation, some 
caution should be used in interpreting the results. However, the T-45 visual simulator 
does appear to have improved the ability of students to fly a proper pattern as they 


approach the aircraft carrier. 
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APPENDIX A. CARRIER QUALIFICATION ATF PERFORMANCE AREAS 


ATF 

Performance Area CQ-1 CQ2x CQ3,4,5,6,7,8,9 CQ-10x CQ-lix 
Headwork | 
Procedures 
Basic Airwork 
Pattern 
Start Position 
Speed Control 
Attitude Control 
Power Control 
Line Up 
Glideslope Control 
Error Detection/Correctio 
Waveoff Technique 
Course Rules 
Pattern Entry/Break 

| Response to LSO 
Progress 
T&G/Bolter Technique 
Formation/Pattern Entry 
Radio Procedures 
Fuel Management 
Deck Procedures 
Catapult Procedures 


PS PK MK MP OO OM 
PoP PP KO KM OM 
PS PS PK PK PS PO PO POO 

PP PK MP PK PO OM OOO 

KP mK MO KM XM MK OK OO 


~*~ 





~~ KK OM 


430 
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APPENDIX B. LSO MARKS AND GRADES 


Definition 

Perfect Pass | 

Reasonable deviations with good corrections 
Fair, reasonable deviations 

Bolter 

Below average but safe pass 

Power waveoff 

Own waveoff 

Waveoff 

Cut. Unsafe, gross deviations inside 


waveoff window 


45 


Grade 


5.0 
40 
3.0 
2.5 
2.0 
2.0 
2.0 
1.0 
0.0 
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APPENDIX C. SUMMARY OF WEATHER OBSERVATIONS 


USS AMERICA (CV 66), May Qualification Period 


Ss — (celsius) 


Date Time 
Z 


1000-1999 
3500-4999 
2000-3499 
2000-3499 
2000-3499 
>8000 
1000-1999 
>8000 
>8000 
>8000 
2000-3499 
2000-3499 
2000-3499 
2000-3499 
2000-3499 
>8000 
2000-3499 
300-599 





Com NS KS RDA WW DANA NWN HA PN 
pK OK —S NK KS WD OH RS = NH LH KD 
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USS JOHN F. KENNEDY (CV 67), July Qualification Period 


Location ind Directior] Wind Speed | Visibility | Temperature 
— = (celsius) 


23-Jul O00! 
0600 

1200 

1800 

24-Jul OOO! 
0600 

1200 

1800 

25-Jul 0001 
0600 

1200 

1800 

0001 

0600 

1200 

1800 

0001 

0600 

1200 

1800 


23-Jul ~—- OOOL 2000-3499 
0600 >8000 
1200 2000-3499 
1800 >8000 
OOO! >8000 
0600 2000-3499 
1200 2000-3499 
1800 2000-3499 
25-Jul OOO1 2000-3499 
0600 2000-3499 
1200 2000-3499 
1800 2000-3499 
26-Jul 0001 2000-3499 
0600 >8000 
1200 1000-1999 
1800 2000-3499 
0001 >8000 
0600 2000-3499 
1200 2000-3499 
1800 2000-3499 





Ww mA NP OD RK HPNUMAN WDA KF KH hh HW HL 
NN WNme NM ON NK NK WN NY NOM & 
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APPENDIX D. CONTINGENCY TABLES FOR COMPARISON OF 
SQUADRON DISQUALIFICATION RATES 


Table D1: Contingency Table for May 
Qualification Period, Both Squadrons. 


Student 

Squadron |} Qualification | Disqualification] Total 
VT-19 4 
had 
4 


0 4 

VT-23 0 3 
Total 80 7 

Chi-Square 0.131 with DF. 


Cell Contents: Count 
Expected Frequency 





Table D2: Contingency Table for July 
Qualification Period, Both Squadrons. 


Student 
Squadron #f Qualification | Disqualification} Total 


36 Z 38 







Chi-Square 0.443 with D-F. ] 





Cell Contents: Count 
Expected Frequency 
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Table D3: Contingency Table for VT-19, 
Both Qualification Periods. 


Student 
Month Qualification | Disqualification] Total 


si 
40.78 3.22 
July 3 2 


0 
6 
Total 
76 
0.441 with D.F. 


Cell Contents: Count 
Expected Frequency 





Table D4: Contingency Table for VT-23, 
Both Qualification Periods. 


Student 


0.143 with D.F. 


Cell Contents: Count 
| Expected Frequency 
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Table D5: Contingency Table for Both Squadrons, Both Qualification 
Periods. 


— May Students July Students Lat 
Squadron — — — ——_ Total 


Cell Contents: Count 
Expected Frequency 





51 


a2 








APPENDIX E. ANALYSIS OF VARIANCE MODEL USED FOR LSO TREND 
ANALYSIS AND CQ-11X PERFORMANCE EVALUATION COMPARISONS 


The same model is assumed for both analyses. Recall that data has been observed 
in each of two months (May, July) for each of two squadrons (VT-19, VT-23). Each 
squadron has its own set of LSO's, assumed to be selected at random from a pool of such 
people; some LSO's participated in both qualification periods, some did not. The number 
of students graded by a given LSO also varies. Letting y,,, represent the /” grade given 


by LSO & in squadron i during month 7. The model states that: 


Vi = Mt a+ B+ (AB) + Yoye + Eur 
where @, = effect due to month /, 
B = effect due to squadron /, 


| 


(ap), = effect of the interaction of month / with squadron /, 
Yun = effect of LSO k (nested within month/squadron, 


combinations), and 


Ex, = effect due to error from grade / by LSO k in 
| squadron i, during month / 
for i = 1,2, 
J = 1,2, 
k = 1...n,, where n, is the number of LSO's in squadron / 
during month /, and 
i = 1... Where n, is the number of grades given by 


LSO & in squadron ij during month /j. 
It is assumed that the a, , & ,(@P), values are unknown constants, or fixed effects, 


while 7,,, are independent, normal (0, c. random effects and &,, are independent, 


normal (0, o°) and mutually independent of the ¥,, random effects. 
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As the number of LSO's varies by squadron and month and the number of students 
_ graded by a given LSO also varies, the data available is unbalanced. The unbalanced 
characteristic of the data causes the estimates of the fixed effects to be nonorthogonal or 
correlated. As a result, the effects of month, squadron, and their interaction, as measured 
by the reduction in the residual sum of squares given by their inclusion in the model, 
depends on the other terms already in the model. In other words, when month enters into 
the model, its effect will depend on the other terms entered into the model before month. 
A different fixed effect may result if month had entered the model first. To compensate 
for the correlation in the data, the significance of these fixed effects is judged by their 
“marginal” reduction in the residual sum of squares, the reduction caused if all other 
terms are also in the model. For example, the adjusted sum of squares for month is the 
difference between the residual sum of squares when the the model considers all variable 
except month and the residual sum of squares when the model considers all variables © 


including month. 
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